Nonlinear Policy Gradient Algorithms for Noise-Action MDPs

We develop a general theory of efficient policy gradient algorithms for Noise-Action MDPs (NMDPs), a class of MDPs that generalize Linearly Solvable MDPs (LMDPs). For finite horizon problems, these lead to simple update equations based on multiple rollouts of the system. We show that our policy gradient algorithms are faster than the PI algorithm, a state of the art policy optimization algorithm. We provide an alternate interpretation of the PIalgorithm that further justifies this: The PI algorithm is actually performing gradient descent with respect to a risk seeking objective, rather than the desired expected cost objective. For infinite horizon problems, we develop algorithms that only require estimation of a state value function, rather than a state-action Q function or advantage function. We develop policy gradient algorithms for all MDP formulations (finite horizon, infinite horizon, first exit) for arbitrary policy parameterizations. We demonstrate the effectiveness of the policy gradient algorithms on simple 2-D nonlinear dynamical systems and large linear dynamical systems.